Replication of Study 1 by Koopmann-Holm et al. (2013, Emotion)

Introduction

Koopmann-Holm et al. investigated how ideal affect (how we want to feel) was affected by the practice of Buddhist-inspired meditation. Following the Affect Valuation Theory (AVT), they hypothesized that “meditators would value calm and other LAP [low arousal positive] states more and excitement and other HAP [high arousal positive] states less than would nonmeditators.”

Study 1 was an online correlation study with the target finding that meditators ranked how often they would ideally like to feel LAP words (“calm” and “peaceful”) significantly higher than nonmeditators when completing the Affect Valuation Index (AVI). This result was the output of an ANCOVA that controlled for age and actual affect. Causal impacts of meditation on ideal LAP were investigated in Koopmann-Holm et al.’s Studies 2 and 3.

Methods

Forty-three university students who had been practicing Buddhist-inspired meditation for at least one year, and 48 students who did not practice any form of meditation were recruited for an online study. Meditators and nonmeditators were recruited separately so that neither group knew to whom they were being compared. Meditators reported meditating for an average of 4.28 years (SD = 2.92), approximately 2–3 times a week (M = 3.28, SD = 1.84, on an 8-point scale, 1 = more than once a day to 8 = never), for about 31.16 minutes per session (SD = 14.39). There were no group differences in gender, % Female: Meditators = 51.16, Nonmeditators = 45.83, chi squared(1, 91) = .26, ns. Meditators were significantly older (mean age = 24.98, SD = 6.49) than Nonmeditators (mean age = 20.90, SD = 1.37), F(1, 89) = 18.12, p < .001, partial eta squared = .17; therefore, we controlled for age in our analyses.

Participants completed an abridged version of the Affect Valuation Index (AVI; Tsai & Knutson, 2006; Tsai, Knutson, & Fung, 2006), which asks participants to use a 5-point rating scale ranging from 1 (never) to 5 (all the time) to rate states that varied in terms of arousal (high, low) and valence (positive, negative) based on the affective circumplex (Feldman Barrett & Russell, 1999; Larsen & Diener, 1992; Watson & Tellegen, 1985). Participants rated how often over the course of a typical week they: (a) actually experienced each state (to assess actual affect) and (b) ideally wanted to feel each state (to assess ideal affect). “Calm” and “peaceful” were used to sample LAP states; “enthusiastic” and “euphoric” were used to sample HAP states; “dull” and “sleepy” to sample LAN states, and “hostile” and “worried” to sample HAN states. Cronbach’s alphas were .85 and .82 for actual and ideal LAP, .74 and .66 for actual and ideal HAP, .63 and .24 for actual and ideal LAN, and .50 and .31 for actual and ideal HAN (alphas for negative states tend to be lower than those for positive states due to restricted variance).

All participants also completed a demographic questionnaire, and meditators completed an assessment of their meditation practice.

Power Analysis

Below is Koopmann-Holm et al.’s “Analyses and Results” discussion for this replication project’s key statistic (in bold).

We initially ran our analyses with Gender as a between subjects factor; however, because there were no significant Gender by Group interactions, we dropped Gender from our final analyses. As mentioned above, we controlled for actual affect when examining group differences in ideal affect, and we controlled for ideal affect when examining group differences in actual affect. The findings did not change when we did not include these covariates.

We conducted separate analyses of covariance (ANCOVAs) for actual and ideal LAP and HAP, controlling for age and the respective other type of affect. As predicted, the main effect of Group (Meditators, Nonmeditators) was significant for ideal LAP [F(1, 87) = 9.34, p < .01, partial eta squared = .10] and ideal HAP [F(1, 87) 4.94, p .05, partial eta squared .05]; meditators valued LAP more and HAP less than did nonmeditators.

The key statistic is an ANCOVA testing whether Meditators and Nonmeditators differ in their ideal LAP, controlling for age and actual LAP [F(1, 87) = 9.34, p < .01, partial eta squared = .10].

A Post Hoc test (Figure 1) reveals that the study had 88% power. The following table of A Priori tests (detailed in Figures 2, 3, 4) show various power and sample sizes to achieve a partial eta squared of 0.10.

A Prior Power Tests

Power (%)	Sample Size
80	73
90	97
95	119

The original effect size was reported as a partial eta squared of 0.10. Below are images from G*Power for a Post Hoc test based on the reported effect size and A Priori tests for 80%, 90%, and 95% power.

Figure 1. G*Power Post Hoc test based on Koopmann-Holm et al., 2013

Figure 2. G*Power A Priori test for 80% power

Figure 3. G*Power A Priori test for 90% power

Figure 4. G*Power A Prior test for 95% power

My target sample size will be just above 80% power (80 usable participant data sets). We will recruit a sufficient number of participants to achieve our planned sample size of 80.

Planned Sample

Those 80 participants will be grouped within the survey as Meditators and Nonmeditators where meditators are defined by Koopmann-Holm et al. as participants “who [have] been practicing Buddhist-inspired meditation [e.g., mindfulness, compassion meditation] for at least one year” and non meditators are defined as those who “did not practice any form of meditation.”

Participants will be recruited through MTurk. The aim is to get AVI data from 40 Nonmeditators and 40 Meditators. I expect that collecting data from 40 Meditators will take longer than collecting Nonmeditator data so the Qualtrics survey contains quotas to cap the number of Nonmeditators allowed to take the AVI portion of the survey at 40. This means that once 40 Nonmeditators have taken the survey there will be additional Nonmeditators who take the non-AVI portion of the survey (and receive 0.10) as I wait for more eligible Meditators to take the AVI. As a result, I will initially create more than 80 HITs on MTurk. See “Procedure” for more details about the survey design.

Materials

Participants completed an abridged version of the Affect Valuation Index (AVI; Tsai & Knutson, 2006; Tsai, Knutson, & Fung, 2006), which asks participants to use a 5-point rating scale ranging from 1 (never) to 5 (all the time) to rate states that varied in terms of arousal (high, low) and valence (positive, negative) based on the affective circumplex (Feldman Barrett & Russell, 1999; Larsen & Diener, 1992; Watson & Tellegen, 1985). Participants rated how often over the course of a typical week they: (a) actually experienced each state (to assess actual affect) and (b) ideally wanted to feel each state (to assess ideal affect). ‘Calm’ and ‘peaceful’ were used to sample LAP states; ‘enthusiastic’ and ‘euphoric’ were used to sample HAP states; ‘dull’ and ‘sleepy’ to sample LAN states, and ‘hostile’ and ‘worried’ to sample HAN states… All participants also completed a demographic questionnaire, and meditators completed an assessment of their meditation practice.

Procedure

I will recruit participants to take a survey through MTurk. The survey collects AVI data from Meditators and Nonmeditators and captures additional relevant information for each group (demographics and/or meditation assessment). The structure of the survey is as follows:

• An IRB consent paragraph and question. Participants may choose to leave the survey and this point (and report any issues that came up).
• A page for the participant to enter their MTurk ID.
• An eligibility block with distractor questions that also triggers an additional assessment at the end of the survey for Meditators. Participants who report meditating but with less than 1 year of experience do not meet the criteria for being either Meditators or Nonmeditators are done with the survey at this point and have the opportunity to report any issues they encountered.
• Participants who meet the criteria to be Meditators or Nonmeditators are then told that they have been selected to continue with the survey for an additional bonus payment of 0.25 if they so choose. Participants who do not complete the AVI receive 0.10.
• The Affect Valuation Index (AVI) items for actual affect used by Koopmann-Holm et al. in the same order but broken up into 3 screen-pages.
• The Affect Valuation Index (AVI) items for ideal affect used by Koopmann-Holm et al. in the same order but broken up into 3 screen-pages.
• A demographic questionnaire.
• Meditation assessment (to determine how frequently participants and how long a typical meditation session lasts).
• A block for the participant to relay any issues that arose during the survey.
• Debriefing.

Each participant will receive $0.35 for completing this entire survey and I plan to collect data from a sufficient number of participants to achieve 40 Nonmeditator and 40 Meditator data sets. The survey can be viewed here. Participants who engage in straight-lining (in this case giving the identical response to an entire page worth of survey questions) will be excluded from analysis.

The original study was conducted using Remark, my replication uses Qualtrics and contains the same AVI words and ordering from Koopmann-Holm et al., 2013.

Analysis Plan

We initially ran our analyses with Gender as a between subjects factor; however, because there were no significant Gender by Group interactions, we dropped Gender from our final analyses. As mentioned above, we controlled for actual affect when examining group differences in ideal affect, and we controlled for ideal affect when examining group differences in actual affect. The findings did not change when we did not include these covariates.

We conducted separate analyses of covariance (ANCOVAs) for actual and ideal LAP and HAP, controlling for age and the respective other type of affect. As predicted, the main effect of Group (Meditators, Nonmeditators) was significant for ideal LAP [F(1, 87) = 9.34, p < .01, partial eta squared = .10] and ideal HAP [F(1, 87) 4.94, p .05, partial eta squared .05]; meditators valued LAP more and HAP less than did nonmeditators.

The main effect of Group, however, was not significant for actual LAP [F(1, 87) = 1.44, ns] or actual HAP [F(1, 87) = 1.23, ns]; meditators and nonmeditators did not differ in how calm or excited they actually felt. The main effect of Group was also not significant for ideal LAN [Meditators: M = 1.09, SE = .04; Nonmeditators: M = 1.11, SE = .03; F(1, 87) = .28, ns], ideal HAN [Meditators: M = 1.08, SE = .04; Nonmeditators: M = 1.10, SE = .03; F(1, 87) = .17, ns], actual LAN [Meditators: M = 1.98, SE = .11; Nonmeditators: M = 1.99, SE = .10; F(1, 87) = .00, ns], or actual HAN [Meditators: M = 1.84, SE = .11; Nonmeditators: M = 1.79, SE = .10; F(1, 87) = .13, ns].

Data will be exported from Qualtrics as a csv file and imported into R. My key analysis of interest will be the ANCOVA for determining difference in LAP ideal affect between Meditators and Nonmeditators controlling for age and LAP actual affect (Study 1, Figure 1, upper left column). This replication will be considered a success if the above ANCOVA reveals that Meditators have higher ideal LAP ratings than Nonmeditators.

Close up on key analysis of interest: ideal LAP for Meditators and Nonmeditators

I will run the ANCOVA with the same covariates: age and respective other type of affect.

I also plan to reproduce the remaining sections of Study 1 Figure 1: Ideal Affect for HAP and Actual Affect findings. In addition, I plan to explore correlations between demographic data (including participants’ general geographic locations with in the US) and AVI responses because some of my future AVI work will compare East and West coast samples (see Exploratory Analyses for more details).

Differences from Original Study

My sample will not be limited to college students, will not be recruited on campus, will not separately recruit Meditators and Nonmeditators (rather they will be identified and branched in the Qualtrics survey), will use Qualtrics instead of Remark, and will focus primarily on replicating the Ideal LAP finding from Study 1. Age differences between a college sample and MTurk sample are the most likely place to anticipate differences in findings.

Methods Addendum (Post Data Collection)

Actual Sample

The results below come from a sample of 85 MTurk participants who took the AVI (after excluding 18 participants): 43 Meditators and 42 Nonmeditators. Average age was 34.7 years old. There were more female than male participants (48 Females, 37 Males). Ethnicity was also reported (51 Caucasian American, 6 Black or African American, 6 Asian American, 1 American Indian or Alaska Native, 1 Asian, 13 Caucasian Europeans, 7 Other). In addition to removing subjects who straight-lined (1), I also removed subjects who left portions of the AVI unanswered (17).

Differences from pre-data collection methods plan

Rather than starting with a batch of 80 HITs as originally planned, I created 8 batches, each with 45 HITs.

Results

Data preparation

I am uploading a version of the data that does not have MTurk IDs or other identifying information (I removed those values from the csv file). NB: I removed the second row of the downloaded csv file which contained the exact text of each question (see .Rmd file for additional information)4.

Following Koopmann-Holm et al.’s analysis, I combine AVI responses for “calm” and “peaceful” together to measure LAP and AVI responses for “enthusiastic” and “euphoric” together to measure HAP.

Confirmatory analysis

Summary Data

Meditator	MeanAge	MeanActualLAP	MeanIdealLAP	ci	cilow	cihi	n
Meditators	35.39535	3.011628	4.360465	0.226710	4.133755	4.587175	43
Nonmeditators	34.00000	3.154762	4.523809	0.208445	4.315365	4.732254	42

ANCOVA Summary

	Df	Sum Sq	Mean Sq	F value	Pr(>F)
Meditator	1	0.57	0.57	1.13	0.2911
Age	1	0.62	0.62	1.23	0.2712
ActualLAP	1	2.35	2.35	4.69	0.0334
Residuals	81	40.67	0.50

Original plot of LAP for Meditators and Nonmeditators

Partial eta squared values

Meditator	0.0137474
Age	0.0149240
ActualLAP	0.0546896

Exploratory analyses

Ideal LAP Responses as Scatterplot

Replicating the other plots in Koopmann-Holm et al. Figure 1

Ideal HAP

Ideal HAP Summary Data

Meditator	MeanAge	MeanActualHAP	MeanIdealHAP	ci	cilow	cihi	n
Meditators	35.39535	2.488372	3.453488	0.2763673	3.177121	3.729856	43
Nonmeditators	34.00000	2.440476	3.464286	0.3013128	3.162973	3.765599	42

Ideal HAP ANCOVA Summary

	Df	Sum Sq	Mean Sq	F value	Pr(>F)
Meditator	1	0.00	0.00	0.00	0.9550
Age	1	2.81	2.81	3.64	0.0598
ActualHAP	1	11.25	11.25	14.56	0.0003
Residuals	81	62.54	0.77

Actual LAP

Actual LAP Summary Data

Meditator	MeanAge	MeanActualLAP	MeanIdealLAP	ci	cilow	cihi	n
Meditators	35.39535	3.011628	4.360465	0.2698924	2.741735	3.281520	43
Nonmeditators	34.00000	3.154762	4.523809	0.2510169	2.903745	3.405779	42

Actual LAP ANCOVA Summary

	Df	Sum Sq	Mean Sq	F value	Pr(>F)
Meditator	1	0.44	0.44	0.60	0.4399
Age	1	0.58	0.58	0.80	0.3748
IdealLAP	1	3.39	3.39	4.69	0.0334
Residuals	81	58.53	0.72

Actual HAP

Actual HAP Summary Data

Meditator	MeanAge	MeanActualHAP	MeanIdealHAP	ci	cilow	cihi	n
Meditators	35.39535	2.488372	3.453488	0.2451104	2.243262	2.733483	43
Nonmeditators	34.00000	2.440476	3.464286	0.2342705	2.206206	2.674747	42

Actual HAP ANCOVA Summary

	Df	Sum Sq	Mean Sq	F value	Pr(>F)
Meditator	1	0.05	0.05	0.10	0.7569
Age	1	4.58	4.58	9.08	0.0035
IdealHAP	1	7.35	7.35	14.56	0.0003
Residuals	81	40.91	0.51

Ideal LAP differences by US Region

ANCOVA Region Summary

	Df	Sum Sq	Mean Sq	F value	Pr(>F)
Region	3	0.14	0.05	0.09	0.9661
Age	1	0.52	0.52	1.00	0.3215
ActualLAP	1	2.47	2.47	4.76	0.0322
Residuals	79	41.08	0.52

Ideal LAP Differences by Reported Ethnicity

## Warning: not plotting observations with leverage one:
##   8, 48

## Warning: not plotting observations with leverage one:
##   8, 48

ANCOVA Ethnicity Summary

	Df	Sum Sq	Mean Sq	F value	Pr(>F)
Ethnicity	6	3.64	0.61	1.21	0.3104
Age	1	0.32	0.32	0.63	0.4305
ActualLAP	1	2.11	2.11	4.19	0.0440
Residuals	76	38.14	0.50

Frequency of Meditators’ practice

Duration of Meditators’ Practice

No linear relationship between Ideal LAP ratings and frequency of meditation practice

No linear increase in Ideal LAP ratings by duration of average meditation session

Discussion

Summary of Replication Attempt

The primary result of the confirmatory analysis shows no significant difference between Meditators’ and Nonmeditators valuation of ideal LAP (as measured by responses to the words “calm” and “peaceful”). Unlike Koopmann-Holm et al. 2013, our MTurk study means show Nonmeditators rating ideal LAP words slightly higher than Meditators. In sum, I failed to replicate the original result.

Commentary

This inability to replicate the key statistic could be explained by a number of differences between the original study and this replication. First, Koopmann-Holm et al. recruited students on campus and then directed them to an online study. That campus population differed in age from our replication and may also contain less variance between participants in comparison with our US-wide sample. Secondly, Koopmann-Holm et al. recruited Meditators and Nonmeditators separately. Our replication used a short block of eligibility questions that masked the eligibility criteria. Thus, while Koopmann-Holm, et al.‘s Meditators were recruited for being meditators (or so it sounds in the article), our Meditators were unaware that their meditation practice was central to the study. Perhaps Koopmann-Holm et al.’s Meditators’ awareness of the importance of their meditating primed them to report higher LAP values? Directly comparing the average number of years Meditators meditated between the original and this replication be ideal. Unfortunately I failed to collect the number of years Meditators practiced (in the final version of the survey participants only noted if it was less than one year or more than one year). While the frequency and duration of meditation looks similar for the original study and the replication, the long term effects (measured in years) remains unknown. Finally, our implementation of the AVI split up ideal and actual affect sections into 3 sections each to minimize participant fatigue. In comparison with the original AVI where ideal and actual affect each have one page, our AVI could cause participants to be less certain of which section (ideal or actual) they were completing (I did, however, repeat the appropriate instructions at the top of each survey page).